A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe
نویسندگان
چکیده
Two of the most fundamental prototypes of greedy optimization are the matching pursuit and FrankWolfe algorithms. In this paper, we take a unified view on both classes of methods, leading to the first explicit convergence rates of matching pursuit methods in an optimization sense, for general sets of atoms. We derive sublinear (1/t) convergence for both classes on general smooth objectives, and linear convergence on strongly convex objectives, as well as a clear correspondence of algorithm variants. Our presented algorithms and rates are affine invariant, and do not need any incoherence or sparsity assumptions.
منابع مشابه
Greedy Optimization and Applications to Structured Tensor Factorizations
Efficiently representing real world data in a succinct and parsimonious manner is of central importance in many fields. We present a generalized greedy framework, which allow us to efficiently solve generalized structured matrix and tensor factorization problems, where the factors are allowed to be from arbitrary sets of structured vectors. Such structure may include sparsity, nonnegativeness, ...
متن کاملGreedy Algorithms for Cone Constrained Optimization with Convergence Guarantees
Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe (FW) algorithms regained popularity in recent years due to their simplicity, effectiveness and theoretical guarantees. MP and FW address optimization over the linear span and the convex hull of a set of atoms, respectively. In this paper, we consider the intermediate case of optimization over the convex cone, parametrized...
متن کاملk-Means Clustering via the Frank-Wolfe Algorithm
We show that k-means clustering is a matrix factorization problem. Seen from this point of view, k-means clustering can be computed using alternating least squares techniques and we show how the constrained optimization steps involved in this procedure can be solved efficiently using the Frank-Wolfe algorithm.
متن کاملFrank-Wolfe Algorithms for Saddle Point Problems
We extend the Frank-Wolfe (FW) optimization algorithm to solve constrained smooth convex-concave saddle point (SP) problems. Remarkably, the method only requires access to linear minimization oracles. Leveraging recent advances in FW optimization, we provide the first proof of convergence of a FW-type saddle point solver over polytopes, thereby partially answering a 30 year-old conjecture. We a...
متن کاملCorrecting Covariate Shift with the Frank-Wolfe Algorithm
Covariate shift is a fundamental problem for learning in non-stationary environments where the conditional distribution ppy|xq is the same between training and test data while their marginal distributions ptrpxq and ptepxq are different. Although many covariate shift correction techniques remain effective for real world problems, most do not scale well in practice. In this paper, using inspirat...
متن کامل